Method and system for classifying image elements

ABSTRACT

A method, system, and machine-readable medium for classifying an image element as one of a plurality of categories, including assigning the image element based on a ratio between an unoccluded perimeter of the image element and an occluded perimeter of the image element and coding the image element according to a coding scheme associated with the category to which the image element is classified. Exemplary applications include image compression, where categories include image foreground and background layers.

REFERENCE TO RELATED DOCUMENTS

The present application is a continuation of U.S. patent applicationSer. No. 13/182,031, filed Jul. 13, 2011, now U.S. Pat. No. 8,244,045,which is a continuation of U.S. patent application Ser. No. 12/638,287,filed Dec. 15, 2009, now U.S. Pat. No. 7,983,491, which is acontinuation of U.S. patent application Ser. No. 11/140,218, filed May27, 2005, now U.S. Pat. No. 7,657,103, which is a continuation of U.S.patent application Ser. No. 10/057,687, filed on Jan. 24, 2002, now U.S.Pat. No. 6,901,169, which claims priority under 35 U.S.C. §119(e) toU.S. provisional application Ser. No. 60/265,544, filed on Feb. 1, 2001,the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates generally to image processing and, moreparticularly, to methods, systems, and machine-readable media forclassifying image elements.

BACKGROUND OF THE INVENTION

Many images are produced using computerized methods that do not rely ona pixel-based representation of an image. Text processing software, forinstance, represents an image using structured page information thatdescribes high-level elements of the image, such as text, fonts, colors,embedded images, etc. This structured page information comes in a largevariety of file formats such as MSWord™ doc, Adobe™ PDF, or PostScript™files. When printed or otherwise rendered, the information may beconverted into a sequence of overlaid image elements that incrementallyconstruct the image.

There is often a need to compress, i.e., encode, these images.Generally, the image elements are first classified as either foregroundor background based on some classification criteria. Afterclassification, the foreground is encoded at a higher resolution becauseit contains the elements of interest. The background, on the other hand,is typically encoded at a lower resolution since it contains elements ofless interest. Such a coding strategy is well known in the art, e.g. inMPEG, JPEG, etc. Thus, the quality of the element classification greatlyaffects the compression ratio and video quality of these images. Assuch, it is important to perform the classification effectively.

Current element classification approaches for images rendered fromstructured page information include classifying all the text in theimage as the foreground and all other details as the background,classifying all the monochrome elements as the foreground and all othersas the background, and classifying the first element drawn as thebackground and all others as the foreground. However, all of theseapproaches are ineffective, particularly for geographical maps, becausethe elements of interest are sometimes rendered such that they meet thecriteria for background classification when, in fact, they areforeground elements. As a result, these elements of interest areerroneously encoded at a lower resolution. As such, the compressionefficiency and video quality of these elements significantly drop.

Accordingly, there is a need in the art for an effective way to classifyimage elements, in general, and image elements rendered from structuredpage information, e.g., electronic documents, in particular.

SUMMARY OF THE INVENTION

The present invention provides a method for classifying an image elementas one of a plurality of categories. The method includes classifying theimage element based on a ratio between an unoccluded perimeter of theimage element and a perimeter of the image element having been occludedby other image elements. The image element thereafter may be codedaccording to a coding scheme associated with the category to which theimage element has been classified.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a system embodying the present invention;

FIG. 1B is a flowchart of an embodiment of a method of the presentinvention;

FIG. 2 is a flowchart of exemplary steps for calculating an imageelement perimeter;

FIG. 3 illustrates an exemplary perimeter calculated according to themethod of FIG. 2;

FIG. 4 is a flowchart of exemplary steps for calculating an occluded andunoccluded perimeter;

FIGS. 5A-5C illustrate exemplary occluded and unoccluded perimeterscalculated according to the method of FIG. 4;

FIG. 6 illustrates the unoccluded perimeter calculation;

FIGS. 7A and 7B are flowcharts of exemplary steps for calculating acolor difference of an image element;

FIG. 8 is a flowchart of another embodiment of a method of the presentinvention;

FIG. 9 illustrates an exemplary electronically produced image decomposedas a sequence of elementary image elements drawn on top of each other,upon which the methods of FIG. 1B or 8 may be applied; and

FIG. 10 is a block diagram of one embodiment of a computer system thatcan implement the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide a technique for classifyingimage elements as one of a plurality of categories based on a ratiobetween an unoccluded perimeter of the element and an occluded perimeterof the element. The occluded perimeter of an image element typicallydiffers from the unoccluded perimeter of the element when other imageelements are overlaid upon it. Thus, the ratio can provide a goodestimate of an amount to which an image element is blocked by otherimage elements. “Pushing” coding errors to highly occluded imageelements can effectively improve perceived quality of a highlycompressed image.

The present invention may be applied to image compression, where imageelements may be classified as belonging to one of foreground andbackground layers. FIG. 1A is a block diagram of an imagecompression_system embodying the present invention. The system includesa rendering engine 10, a classifier 20, a foreground coder 30, and abackground coder 40. The rendering engine 10 generates the image to becompressed. The rendering engine 10 can be, but is not limited to, aprint driver or text processing software. The classifier 20 processesthe image by classifying its image elements as either foreground orbackground elements. The elements classified as foreground may be codedaccording to any coding technique by the foreground coder 30. Similarly,the elements classified as background may be coded according to anycoding technique by the background coder 40. Exemplary coding techniquesare well known in the art. Such techniques may be used independently orin conjunction with coding techniques described in “High QualityDocument Image Compression with DjVu”, by L. Bottou, et. al., Journal ofElectronic Imaging, 7(3): 410-425, 1998, for example. Typically, theforeground and background coding techniques will be selected to providecompression of the foreground image elements with low loss of imagequality. Whereas, the background image elements will be coded withhigher compression and relatively higher loss of image quality.

It is to be understood that the image compression application is forexemplary purposes only, as the present invention may also be used inany application where image element classification is performed. It isto be further understood that the number of categories is not limited totwo.

In exemplary image compression, a low ratio between unoccluded andoccluded perimeters of an image element may indicate that if the elementwere classified as belonging to the background layer, any coding errorswith respect to the element's boundaries are likely to be obscured byother occluding elements from the foreground layer. Thus, the codingerrors are unlikely to be noticed. By contrast, a high ratio betweenunoccluded and occluded perimeters of the image element may indicatethat coding errors with respect to the boundaries are likely to beobserved because they will not be obscured by other elements. In thisevent, it may be appropriate to code the element in the foregroundlayer. Thus, the present invention provides a higher compression ratioand improved visual quality of compressed images.

The number of bits for encoding foreground elements may be proportionalto the perimeter P_(occluded) of the visible part of the element (i.e.,after removing the element shape portions that are occluded by otherelements). The edges of background elements that arise from occlusionsby foreground elements may be defined by the boundary of the occludingforeground elements. So, the number of bits for encoding backgroundelements may be reduced by excluding the occluded parts of thebackground elements, which may be already encoded as part of theforeground elements. Thus, the number of bits for encoding backgroundelements may be proportional to the length P unoccluded of the perimetersegments that do not result from occlusions by foreground elements.Furthermore, the proportionality coefficient depends on the colordifferences along the element boundary.

FIG. 1B is a flowchart of a method according to an embodiment of thepresent invention. According to the invention, an image classificationsystem calculates an unoccluded perimeter of an image element,P_(unoccluded), (100) and a perimeter of the image element when occludedby other image elements, P_(occluded) (110). The system compares a ratioof these calculated values to a predetermined threshold T (120):

$\begin{matrix}{\frac{P_{unoccluded}}{P_{occluded}} > {T.}} & (1)\end{matrix}$If the ratio of P_(unoccluded) to P_(occluded) exceeds the threshold,then the system classifies the image element as belonging to theforeground layer (140). Otherwise, the system classifies the imageelement as belonging to the background layer (130). An exemplarythreshold T is 80%. The system then codes the image element using eithera foreground or background coding scheme based on the elementclassification (150).

Optionally, the system assigns some predetermined types of imageelements to either the foreground or background layer. For example, thesystem may pre-assign text and symbols to the foreground layer. Sincethe foreground may be encoded with higher resolution, encoding text andsymbols as foreground improves their readability.

It is to be understood that all or portions of the present invention maybe used in connection with other methods for classifying image elements,as some aspects may be optional. For example, if text may beautomatically assigned to the foreground layer, the method need not beapplied to textual image elements.

An image element processed according to the present invention may beinitially represented as structured page information which describes thecomponents of the image element and its coordinates in an image. Most ofthese components are simple operations such as “fill a rectangle,” “drawa line,” or “draw a piece of text.” These operations simply assign asolid color to a set of specified portions called the element shape.Later operations may overwrite image data generated by earlieroperations. When the printing operations are completed, they render acomplete printed output (often a page).

In the present invention, the system processes the image element fromits structured page information, such that the image element coordinatesare read and compared to foreground and background layers. From thiscomparison, the system determines the occluded portions of the imageelement. Then, the system renders the image element into either theforeground or background layers.

FIG. 2 is a flowchart of an exemplary method for calculating the imageelement perimeter. The method makes a number of Boolean operationsbetween the foreground and background layers and the image elementshapes. In this embodiment, these layers and shapes may be representedusing run-length encoding. In run-length encoding, each scan line (orrow) of the layers and image elements is represented by a sequence oflengths that describe successive runs of black and white bits. A run isa contiguous group of 0's (white bits) or 1's (black bits) encounteredin a left to right scan of the scan line. For exemplary purposes, theblack bits represent image elements and the white bits representnon-element space. Accordingly, the system calculates the perimeter of arun-length-encoded layer or element shape by making a single pass on the“black bit” runs, scanning each scan line of the layer or element shapefrom left to right (200).

For each “black bit” run, the system calculates the run's perimeter r byadding twice the length of the run, i.e., the number of bits, and twicethe width (205). The values of the length and width are doubled toaccount for, respectively, the top and bottom lengths and the endwidths. Then, the system calculates the length l of each contact segmentbetween the run and the adjacent run in the next scan line, i.e., therun bits in the next scan line that are adjacent to the run and are partof the same layer or element shape (210).

After processing all the runs, the system sums all the perimeters r toproduce R and all the contact lengths 1 to produce L (220). Sinceperimeters of adjacent runs include the same contact length, the systemmultiplies the sum L by two to account for the duplicate inclusion. Thesystem calculates the perimeter of the layer or element shape as P=R−2L(230).

FIG. 3 illustrates the perimeter calculation in FIG. 2 for an exemplaryimage element. In this example, the image element 300 is made up of 5scan lines. Each scan line has a single “black bit” run, with theexception of the third scan line, which has two, separated by a “whitebit” run. The system determines the perimeter r of each “black bit” run310. Then, the system computes the sum R of the perimeters r of the runs310. The system detects the contact segments 320 between adjacent runs,sums all their lengths l, and multiplies the sum by 2 to produce 2L. Theperimeter P of the image element 300 equals R-2L.

It is to be understood that the perimeter calculation is for exemplarypurposes only, as the perimeter calculated by other techniques may alsobe used by the present invention. Such techniques include, but are notlimited to, contour-mapping and region-growing.

FIG. 4 illustrates an exemplary method for detecting the unoccluded andoccluded portions of the image elements for which the system calculatesperimeters using the method of FIG. 2, for example, such that the imageelement may be classified as belonging to either foreground orbackground layers. First, the system creates two empty layers F and B torepresent the image elements to be classified as foreground andbackground. The system then performs the following on every imageelement starting from the topmost element and proceeding toward thebottommost element.

The system calculates the perimeter P_(original) of the image element(400). This is the perimeter of the original shape of the image elementas it would appear without occlusion. Then, the system determines theportions of the image element shape occluded by background imageelements drawn above the current image element (405). This is achievedby calculating the intersection of the image element shape and thecurrent background B. For example, for a given image element portion i,if the current background portion B(i) has a value, then the imageelement portion i is designated as occluded and removed.

For the first image element processed, i.e. the topmost element, thebackground B is empty, such that there are no occluded portions toremove. For subsequent elements, if the background includes previouslyprocessed occluding elements, the system removes the occluded portionsfrom the image element shape (410).

Next, the system determines the portions of the resulting element shapeoccluded by foreground image elements drawn above the current imageelement (420). This is achieved by calculating the intersection of theimage element shape and the current foreground F. For example, for agiven image element portion i, if the current foreground portion F(i)has a value, then the image element portion i is designated as occludedand removed.

For the first image element processed, i.e., the topmost element, theforeground F is empty, such that there are no occluded portions toremove. For subsequent elements, if the foreground includes previouslyprocessed occluding elements, the system removes the occluded portionsfrom the image element shape (425). The system then calculates theperimeter P_(occlusion) of the portions occluded by foreground andbackground elements (430).

The resultant image element shape includes only the visible portions ofthe image element. The system now calculates the occluded perimeterP_(occluded) of the element and the unoccluded perimeter P_(unoccluded)of the element (440), whereP _(occluded) =R−2L,  (2)the perimeter of the visible portions, as described previously. And,

$\begin{matrix}{{P_{unoccluded} = \frac{P_{original} + P_{occluded} - P_{occlusion}}{2}},} & (3)\end{matrix}$the perimeter of the visible boundary, where the sum of the occluded andoriginal perimeters is equal to twice the unoccluded perimeter of theimage element plus the perimeter of the occluded portions that areremoved.

FIGS. 5A-5C illustrate exemplary occluded and unoccluded portions of animage element detected according to a method of the present invention.FIG. 5A illustrates a polygonal image element 700 which is occluded by‘ab’ image element 710. The occluded perimeter P_(occluded) of thepolygon 700, i.e., the perimeter of the visible portions with theoccluded portions removed, is shown in FIG. 5B. The unoccluded perimeterP_(unoccluded) of the polygon 700, i.e., the discontinuous visibleboundaries (indicated by thick black lines), is shown in FIG. 5C.

FIG. 6 illustrates the calculation from Equation (3) used to determinethe unoccluded perimeter of the polygon 700. The sum of the original andoccluded perimeters, P_(original) and P_(occluded), is equal to twicethe unoccluded perimeter P_(unoccluded) plus the perimeter of theoccluded portions P_(occlusion).

It is to be understood that the method for detecting unoccluded andoccluded portions of the image elements is for exemplary purposes only,as many variations of this method may be used by the present invention.

Optionally, a color difference δ may be included in the ratio,

$\begin{matrix}{\frac{\delta\; P_{unoccluded}}{P_{occluded}} > {T.}} & (4)\end{matrix}$The color difference may be used to determine how closely the color ofan image element matches that of the background. If the colors areclosely matched, then the edges of the image element are not verydistinct, such that the likelihood of the element belonging to thebackground is high.

Generally, multiple passes over an image element are used to determinewhether the element belongs in the foreground or background. This isdone because elements located below the current element may turn out tobe foreground elements, rather than background elements, and may thusaffect the perimeter of the current element. In the present invention,the calculation of the color difference allows an estimate of thelikelihood of the current element being assigned to background, thusenabling a single pass over an image element to determine whether itshould be foreground or background.

Since the computation of the color difference may be very expensive, twoexemplary simplifications may be used. FIG. 7A is a flowchart of a firstexample. The system calculates the color difference between the elementboundary and each of the adjacent background elements (500). Then, thesystem selects the largest of these color differences as δ (510).

FIG. 7B illustrates a second example. The system selects a predeterminednumber of background elements (550). The system heuristically determinesthese background elements during a preliminary pass over the imageelements according to the background elements' overall size and form.That is, an image element, such as a solid rectangle, with a large areacompared to its bounding box (i.e., an imaginary boundary around theelement) and a large area overlapping with the current image element isa potential background element. As such, the image element may be usedto calculate the color difference. Conversely, an image element, such asa long thin curve, with a small area compared to its bounding box and asmall area overlapping with the current image element is not a potentialbackground element. As such, the image element may not be used tocalculate the color difference.

After the system selects the background elements, the system calculatesthe color difference between each of the selected adjacent backgroundelements and the element boundary (560). The system then averages thecolor differences as δ (570).

It is to be understood that the methods described for calculating colordifference are for exemplary purposes only, as many techniques tocalculate color difference may be used in the present invention. Suchtechniques include, but are not limited to, gradient measurements.

FIG. 8 is a flowchart of a method according to another embodiment of thepresent invention. According to the invention, an image classificationsystem initializes a foreground layer F and a background layer B toempty (580). Then, the system iteratively classifies each image element(582).

For each image element, the system performs the following. The systemdetermines the portions of the image element that intersect with theforeground and background layers (583), then identifies the intersectingportions as occluded portions of the image element and removes them(584). The system then calculates an unoccluded perimeter of the imageelement, P_(unoccluded), (585) and an occluded perimeter of the imageelement, P_(occluded) (586). The system compares a ratio of thesecalculated perimeters to a predetermined threshold T (587), as inEquation (1). If the ratio of P_(unoccluded) to P_(occluded) exceeds thethreshold, then the system classifies the image element as belonging tothe foreground layer (589). Otherwise, the system classifies the imageelement as belonging to the background layer (588). An exemplarythreshold T is 80%. Optionally, the system includes a color differencein the ratio, as in Equation (4).

After all the image elements are classified, the system then codes theforeground and background layers using their respective coding schemes(590).

FIG. 9 shows an exemplary electronically produced image, which may besegmented according to the present invention. The system decomposes theimage 600 into a sequence of elementary image elements 610 through 655drawn on top of each other, where the front 610 and the back 655 of thesequence indicate the image elements that are topmost and bottommost inthe sequence, respectively. That is, the system prints the front imageelement last and the back image element first. The system may print theelementary image elements 610 through 655 separately as illustrated inFIG. 9 using printing software that processes the given image format.

According to the present invention, the system first prepares emptyforeground and background layers F and B representing the image elementsto be classified as foreground and background. Then, the systemprocesses the first element 610 according to the method of the presentinvention. That is, the system calculates the perimeter P_(original) ofthe original shape of the first element using the perimeter calculationof FIG. 2, for example. Then, the system determines the portions of thefirst element shape that are occluded by background and foreground imageelements drawn above the current image element as in FIG. 4, forexample. Since the layers F and B are empty at this point, none of thefirst element is occluded and none of its portions are removed. So, theperimeter P_(occlusion) of these non-existent occluded portions is nil.

Now, the system computes the perimeters P_(occluded) and P_(unoccluded)of the first element from Equations (2) and (3). Optionally, the systemmay calculate the color difference between the first element and thebackground according to either method of FIG. 7A or 7B, for example.Then, the system calculates the ratio of the occluded and unoccludedperimeters. Based on the ratio, the system classifies the first elementas belonging to either the foreground or background layer, as shown inFIG. 1B, for example. Since the first element is unoccluded, theoccluded and unoccluded perimeters are the same. Thus, the first elementhas a ratio of 1 or, with the color difference, higher. If the thresholdis set at 80% (i.e., 0.8), then the first element is assigned to theforeground. The system then updates the foreground layer F to includethe first element.

In the case of an occluded image element 650, which is occluded byelements 610, 615, 620, 630, and 635, the layers F and B may not beempty because elements 610 through 645 will have been processed andclassified prior to element 650. Thus, the system determines and removesthe occluded parts of the element 650. The occluded perimeterP_(occluded) of the element 650 may be larger than the unoccludedperimeter P_(unoccluded). As such, the ratio between the perimeters maybe lower than the threshold. In which case, the system classifies theelement 650 as background.

The system repeats the method of the present invention for the imageelements 610 through 655, resulting in classified image elements, whichthe system then efficiently encodes.

The mechanisms and methods of the present invention may be implementedusing a general-purpose microprocessor programmed according to theteachings of the present invention. The present invention thus alsoincludes a machine-readable medium which includes instructions which canbe executed by a processor to perform a method in accordance with thepresent invention. This medium can include, but is not limited to, anytype of disk including floppy disk, optical disk, CD-ROMs, or any typeof media suitable for storing electronic instructions.

FIG. 10 is a block diagram of one embodiment of a computer system thatcan implement the present invention. The system 700 may include, but isnot limited to, a bus 710 in communication with a processor 720, asystem memory module 730, and a storage device 740 according toembodiments of the present invention.

It is to be understood that the structure of the software used toimplement the invention may take any desired form. For example, themethod illustrated may be implemented in a single program or multipleprograms.

Numerous modifications and variations of the present invention arepossible in light of the above teachings. It is therefore to beunderstood that within the scope of the appended claims, the inventionmay be practiced otherwise than as specifically described herein.

What is claimed is:
 1. A method comprising: determining a relativeocclusion between a first image element of an image and a second imageelement of the image, to yield an occluded image element; assigning theoccluded image element to an occluded category; and processing, via aprocessor, the image based on the occluded category.
 2. The method ofclaim 1, further comprising using a perimeter ratio to determine therelative occlusion.
 3. The method of claim 1, further comprising using acolor difference to determine the relative occlusion.
 4. The method ofclaim 3, wherein the color difference is calculated using a largestcolor difference between the first image element and the second imageelement.
 5. The method of claim 1, wherein the occluded image element isassigned to the occluded category upon the relative occlusion meeting apredetermined threshold.
 6. The method of claim 1, further comprisingdetermining a first shape of the first image element and a second shapeof the second image element.
 7. The method of claim 1, furthercomprising generating an output image based on the processing of theimage.
 8. A system comprising: a processor; and a computer-readablestorage medium storing instructions which, when executed by theprocessor, cause the processor to perform a method comprising:determining a relative occlusion between a first image element of animage and a second image element of the image, to yield an occludedimage element; assigning the occluded image element to an occludedcategory; and processing the image based on the occluded category. 9.The system of claim 8, the computer-readable storage medium storingadditional instructions which result in the method further comprisingusing a perimeter ratio to determine the relative occlusion.
 10. Thesystem of claim 8, the computer-readable storage medium storingadditional instructions which result in the method further comprisingusing a color difference to determine the relative occlusion.
 11. Thesystem of claim 10, wherein the color difference is calculated using alargest color difference between the first image element and the secondimage element.
 12. The system of claim 8, wherein the occluded imageelement is assigned to the occluded category upon the relative occlusionmeeting a predetermined threshold.
 13. The system of claim 8, thecomputer-readable storage medium storing additional instructions whichresult in the method further comprising determining a first shape of thefirst image element and a second shape of the second image element. 14.The system of claim 8, the computer-readable storage medium storingadditional instructions which result in the method further comprisinggenerating an output image based on the processing of the image.
 15. Acomputer-readable storage medium storing instructions which, whenexecuted by a computing device, cause the computing device to perform amethod comprising: determining a relative occlusion between a firstimage element of an image and a second image element of the image, toyield an occluded image element; assigning the occluded image element toan occluded category; and processing the image based on the occludedcategory.
 16. The computer-readable storage medium of claim 15 storingadditional instructions which result in the method further comprisingusing a perimeter ratio to determine the relative occlusion.
 17. Thecomputer-readable storage medium of claim 15 storing additionalinstructions which result in the method further comprising using a colordifference to determine the relative occlusion.
 18. Thecomputer-readable storage medium of claim 17, wherein the colordifference is calculated using a largest color difference between thefirst image element and the second image element.
 19. Thecomputer-readable storage medium of claim 15, wherein the occluded imageelement is assigned to the occluded category upon the relative occlusionmeeting a predetermined threshold.
 20. The computer-readable storagemedium of claim 15, storing additional instructions which result in themethod further comprising determining a first shape of the first imageelement and a second shape of the second image element.